{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tutorial TensorFlow Eager.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "FzI7Rr_sK8lN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "This notebook introduces the **eager execution** for TensorFlow, a low-level interface allowing a more dynamic programming experience. Eager execution greatly simplifies how you can write and debug models, softening the complete separation between the *definition* of the operations and their *execution* in the standard TensorFlow interface."
      ]
    },
    {
      "metadata": {
        "id": "JQPc5c66LBJf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Table of contents:"
      ]
    },
    {
      "metadata": {
        "id": "zi6BJup8sKQy",
        "colab_type": "toc"
      },
      "cell_type": "markdown",
      "source": [
        ">[Static and dynamic graph computation](#scrollTo=JOX81EBYEfJ6)\n",
        "\n",
        ">[Enabling eager execution](#scrollTo=qAFSauM9nKwN)\n",
        "\n",
        ">>[Additional: experimenting with eager execution in the nightly branch](#scrollTo=kBNAHEBU9Uvj)\n",
        "\n",
        ">[A neural network with eager execution](#scrollTo=5P0ToXGupvlu)\n",
        "\n",
        ">>[Loading data](#scrollTo=tSheSH8ArO4N)\n",
        "\n",
        ">>[Variables and gradients with eager execution](#scrollTo=8azE_xwVr4ak)\n",
        "\n",
        ">>>[Eager variables and NumPy compatibility](#scrollTo=2lkQTVjyiHN-)\n",
        "\n",
        ">>>[Computing gradients in eager](#scrollTo=VtfqaWPyiLJG)\n",
        "\n",
        ">>[Defining models using layers](#scrollTo=MZL5tCFWPaat)\n",
        "\n",
        ">>[Handling the input pipeline](#scrollTo=kfB5YWC2xnus)\n",
        "\n",
        ">>[Gradients' computation and model's optimization](#scrollTo=ehLIBkN23ltL)\n",
        "\n",
        ">>[Evaluating the model and plotting the results](#scrollTo=XecSJK4r9PD1)\n",
        "\n",
        ">>>[Computing and visualizing metrics](#scrollTo=5ix3nr8wqglP)\n",
        "\n",
        ">>>[Saving summaries on disk and working with the TensorBoard](#scrollTo=jlZIDj8VqjRz)\n",
        "\n",
        ">[Advanced features](#scrollTo=C9PxnV3x_EJo)\n",
        "\n",
        ">>[Enabling GPU execution](#scrollTo=rdTN7c_k_GsN)\n",
        "\n",
        ">>[Saving and reloading](#scrollTo=b_I-_yQnB_Cw)\n",
        "\n",
        ">[Final remarks](#scrollTo=fVtZn99KDDSG)\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "JOX81EBYEfJ6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Static and dynamic graph computation"
      ]
    },
    {
      "metadata": {
        "id": "ZX1Ad2mcEsE4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To understand the need for eager execution, consider a simple example:"
      ]
    },
    {
      "metadata": {
        "id": "bOZtwTAcEbHc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5abd08f1-eab7-42f4-ed1e-a5b4286325eb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521032153786,
          "user_tz": -60,
          "elapsed": 7543,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "a = tf.constant(3.0)\n",
        "b = a + 2.0\n",
        "print(b)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tensor(\"add:0\", shape=(), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lQatWScWHiNh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If you never played with the low-level components of TensorFlow before, you probably would have expected the print operation to show the value of `b` at this point. Instead, we have to fetch the value of the variable by running the operation inside a `Session` object:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "rwhQ5L1zGi17",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2750d9bb-0bb5-457a-d14c-3da53ad02308",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521032156114,
          "user_tz": -60,
          "elapsed": 1073,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "sess = tf.Session()\n",
        "with sess.as_default():\n",
        "  print(sess.run(b))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zr-L0vKiIoz-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, let's keep exploring and add some intermediate computations:"
      ]
    },
    {
      "metadata": {
        "id": "Bre_99c5HdaS",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1d625c80-b068-44e9-e3a1-b7b89f836777",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521032158722,
          "user_tz": -60,
          "elapsed": 521,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.Session() as sess:\n",
        "    c = sess.run(1.5*b)\n",
        "print(b)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tensor(\"add:0\", shape=(), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "2dx72nlWI6HA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once again, you might have expected to obtain the value of ```b```, since it was clearly computed to obtain ```c```. While there are several methods to get it (such as calling its ```eval``` method), none of them is as satisfying as writing pure NumPy code, and none of them lends itself to an immediate debug using a visual editor. \n",
        "\n",
        "Now consider the same code written with the [eager execution](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/eager) enabled:\n",
        "\n",
        "```python\n",
        "a = tf.constant(3.0)\n",
        "b = a + 2.0\n",
        "print(b) # print tf.Tensor(5.0, shape=(), dtype=float32)\n",
        "```\n",
        "\n",
        "We can now get the value immediately! With eager execution enabled, definition and execution are tied, with one following the other automatically. Let us see how to enable it before moving on to more interesting examples."
      ]
    },
    {
      "metadata": {
        "id": "qAFSauM9nKwN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Enabling eager execution"
      ]
    },
    {
      "metadata": {
        "id": "ysEhDOAdncOI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Eager was introduced experimentally in TensorFlow v1.5, but in order to use all functionalities, we need to install the latest version (v1.7rc0 as of this writing). To upgrade from terminal (replace **tensorflow** with **tensorflow-gpu** for the version with GPU support):"
      ]
    },
    {
      "metadata": {
        "id": "2eEUo9EjjhzW",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 15
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "outputId": "24180439-88c8-45e7-9fe3-6e0ef052aa88",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521042835031,
          "user_tz": -60,
          "elapsed": 24533,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install tensorflow==v1.7rc0"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting tensorflow==v1.7rc0\n",
            "  Downloading tensorflow-1.7.0rc0-cp36-cp36m-manylinux1_x86_64.whl (48.1MB)\n",
            "\u001b[K    100% |████████████████████████████████| 48.1MB 29kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: tensorboard<1.7.0,>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: protobuf>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.7.0,>=1.6.0->tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: bleach==1.5.0 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.7.0,>=1.6.0->tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.7.0,>=1.6.0->tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: html5lib==0.9999999 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.7.0,>=1.6.0->tensorflow==v1.7rc0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.4.0->tensorflow==v1.7rc0)\n",
            "Installing collected packages: tensorflow\n",
            "  Found existing installation: tensorflow 1.6.0\n",
            "    Uninstalling tensorflow-1.6.0:\n",
            "      Successfully uninstalled tensorflow-1.6.0\n",
            "Successfully installed tensorflow-1.7.0rc0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4uG9v_14npop",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Eager is enabled with a single line:"
      ]
    },
    {
      "metadata": {
        "id": "jfP0tOofnJ9S",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 88
        },
        "outputId": "8a13b775-a319-4222-94a9-50634ed3eee2",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043186281,
          "user_tz": -60,
          "elapsed": 5203,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow.contrib.eager as tfe\n",
        "import numpy as np\n",
        "tf.enable_eager_execution()"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use the retry module or similar alternatives.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_8zJc4-Vpd3A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The previous instruction should always be run at the beginning of a program. If it fails, simply reset the runtime of the notebook from Runtime >> Restart runtime. If you are working with v1.5 or v1.6, replace `tf.enable_eager_execution()` with `tfe.enable_eager_execution()`"
      ]
    },
    {
      "metadata": {
        "id": "kBNAHEBU9Uvj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Additional: experimenting with eager execution in the nightly branch"
      ]
    },
    {
      "metadata": {
        "id": "C_Nl4k519hXg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Since eager is under development, new features are being constantly included (e.g., additional layers) and are not immediately available on the main branch. If you want to experiment with the latest eager module, the recommended way is to setup a [virtual environment](https://www.tensorflow.org/install/install_linux#InstallingVirtualenv) and install the nightly branch of TF."
      ]
    },
    {
      "metadata": {
        "id": "FLYGPVWp-6qs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The virtual environment can be created using either [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) or the [Anaconda distribution](https://conda.io/docs/user-guide/tasks/manage-environments.html), and the installation is specific to the OS. For example, using virtualenv on a Unix platform we can create and activate a new environment as follows: "
      ]
    },
    {
      "metadata": {
        "id": "MuFXS1w_-5QT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "```python\n",
        "virtualenv --system-site-packages -p python3 tfeager\n",
        "source ~/tfeager/bin/activate\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "UODpDBXdA0KX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The [official installation guide of TensorFlow](https://www.tensorflow.org/install/) describes the proper commands for other systems as well. Once a clean virtual environment is running, you can install the nightly build of TensorFlow as a standard Python package:"
      ]
    },
    {
      "metadata": {
        "id": "av0F96FcZur5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "  ```\n",
        "  !pip install tf-nightly # replace with tf-nightly-gpu for GPU support\n",
        "  ```"
      ]
    },
    {
      "metadata": {
        "id": "5P0ToXGupvlu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# A neural network with eager execution"
      ]
    },
    {
      "metadata": {
        "id": "tSheSH8ArO4N",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading data\n",
        "\n",
        "We are going to code a simple neural network on the [Iris dataset](https://archive.ics.uci.edu/ml/datasets/iris) to highlight some differences with the standard TF execution. For the purpose of this demo, we will load the scikit-learn version of Iris, preprocess it, and split it into two sets:"
      ]
    },
    {
      "metadata": {
        "id": "m8tCHLpWp0MD",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "from sklearn import datasets, preprocessing, model_selection\n",
        "data = datasets.load_iris()\n",
        "# Feature normalization on the input\n",
        "X = preprocessing.MinMaxScaler(feature_range=(-1,+1)).fit_transform(data['data'])\n",
        "# Encode the output using the one-hot encoding\n",
        "y = preprocessing.OneHotEncoder(sparse=False).fit_transform(data['target'].reshape(-1, 1))\n",
        "# Split in train/test sets\n",
        "X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.25, stratify=y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8azE_xwVr4ak",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Variables and gradients with eager execution"
      ]
    },
    {
      "metadata": {
        "id": "2lkQTVjyiHN-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Eager variables and NumPy compatibility"
      ]
    },
    {
      "metadata": {
        "id": "X1REd6l0szLs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Before going straight to the model definition, let us see how variables and gradients are handled under the eager execution. First, eager comes with its own implementation of variables, which are automatically initalized when requested:"
      ]
    },
    {
      "metadata": {
        "id": "Pbq_WeeBr1Jk",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "67814045-72e8-4cd5-db48-e9fa76e60d5f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043211471,
          "user_tz": -60,
          "elapsed": 515,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "W = tfe.Variable(0.5, name='w')\n",
        "print(W)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<tf.Variable 'w:0' shape=() dtype=float32, numpy=0.5>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Rhh_2-yRHSXw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To further simplify development, in eager execution you can mix TF Tensors and NumPy arrays with automatic casting underneath:"
      ]
    },
    {
      "metadata": {
        "id": "YLymy2ciHGRW",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f9b3797d-45b7-419d-a6c3-5208603ce157",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043213607,
          "user_tz": -60,
          "elapsed": 516,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print(W + np.asarray([1, 3]))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor([1.5 3.5], shape=(2,), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "fGrcBXc_IPfn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "And you can extract the NumPy representation of a Tensor with a new `numpy` method:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "J-d5d6usIbbd",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "56dd2b4d-65b8-4a43-8e17-6a84c653efd4",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043234204,
          "user_tz": -60,
          "elapsed": 503,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "W.numpy()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "metadata": {
        "id": "_is0UUtiDZ83",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finally, we can initialize an eager variable using the standard `get_variable` method, by specifying a flag:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "JbY5E970DjY4",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "93864f82-5b4a-4967-fd3c-450bdb2b2bac",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043236078,
          "user_tz": -60,
          "elapsed": 545,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "W2 = tf.get_variable('W2', shape=[1], use_resource=True)\n",
        "print(W2)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<tf.Variable 'W2:0' shape=(1,) dtype=float32, numpy=array([0.5054914], dtype=float32)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VtfqaWPyiLJG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Computing gradients in eager"
      ]
    },
    {
      "metadata": {
        "id": "Bzu1WIiYM9X-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Gradients computation is where the real simplicity the eager execution comes to fruition. When eager execution is enabled, functions can be defined using standard Python syntax, and the `tfe.gradients_function` will return the handle to another *Python* function to compute the gradient:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "F3M7DqsgI0YG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "53825ccd-2273-4b8b-c2e4-bc3240a2e6eb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043238253,
          "user_tz": -60,
          "elapsed": 569,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def f(X):\n",
        "  return 1.0 + 2.0 * X\n",
        "f_grad = tfe.gradients_function(f)\n",
        "print(f_grad(0.3))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<tf.Tensor: id=43, shape=(), dtype=float32, numpy=2.0>]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "LRuBWsdSintp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "In programming terms, we are now dealing with an **imperative** definition, as opposed to a **declarative** interface.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "b2XLHW1sN-pr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can also use `tfe.value_and_gradients_function` to compute the value and the gradients simultaneously. By default, the gradient is computed with respect to all the parameters of the function. We can specify a list of parameters explicitly, either by name or by indexing:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "scYI36DINxId",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "50b521c4-673c-4e66-942a-58f698e490d8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043240048,
          "user_tz": -60,
          "elapsed": 520,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "a = tf.constant(0.3)\n",
        "b = tf.constant(0.5)\n",
        "\n",
        "def f(a, b):\n",
        "  return a*b\n",
        "\n",
        "# Return the gradient for the first parameter only\n",
        "print(tfe.gradients_function(f, params=[0])(1.0, 1.0))\n",
        "\n",
        "# Alternative definition (by name)\n",
        "# print(tfe.gradients_function(f, params=['a'])(1.0, 1.0))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[<tf.Tensor: id=48, shape=(), dtype=float32, numpy=1.0>]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8DEYc1fFO4hO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "And we can chain the use of `tfe.gradients_function` to get higher-order derivatives:"
      ]
    },
    {
      "metadata": {
        "id": "TMRDPnQmO-87",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7f3bf4e7-a2e2-4927-b36f-d44e3f9e7e49",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043241713,
          "user_tz": -60,
          "elapsed": 630,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def f(X):\n",
        "  return tf.square(X)\n",
        "\n",
        "# Second-order derivative\n",
        "f_gg = tfe.gradients_function(tfe.gradients_function(f))\n",
        "f_gg(1.0)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<tf.Tensor: id=58, shape=(), dtype=float32, numpy=2.0>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "MZL5tCFWPaat",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Defining models using layers"
      ]
    },
    {
      "metadata": {
        "id": "AKFjb1T0uRD6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "While we can use variables to directly define our model, this is clearly unpractical as soon as the model starts to get complicated. To simplify the development with eager execution enabled, there are a number of strategies we can adopt."
      ]
    },
    {
      "metadata": {
        "id": "f8KcSfvyjNM_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, we can use all the blocks defined in the [layers' module](https://www.tensorflow.org/api_docs/python/tf/layers), which automatically detect we are working with eager enabled and initialize their variables accordingly. For example, this is a simple linear model:"
      ]
    },
    {
      "metadata": {
        "id": "Qzw-u4vGt-wu",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "lin = tf.layers.Dense(units=3, use_bias=True, activation=None)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3aSnJmEiEvTb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can stack multiple layers to create more complicated models. For example, a neural network with one hidden layer and dropout in the middle:"
      ]
    },
    {
      "metadata": {
        "id": "uqhgB4bXEup1",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "hid = tf.layers.Dense(units=10, activation=tf.nn.relu)\n",
        "drop = tf.layers.Dropout()\n",
        "out = tf.layers.Dense(units=3, activation=None)\n",
        "\n",
        "def nn_model(x, training=False):\n",
        "  return out(drop(hid(x), training=training))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8i-dVMxfjbDg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note the `training` flag we can use to differentiate between training and test (for the dropout layer). This is a standard practice for all layers that have a different behaviour in the two cases.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "nfueqIQ7vBjM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For more complicated models, it becomes useful to have an object-oriented interface to keep track of how the layers are organized. TF provides such a functionality with the [Model](https://www.tensorflow.org/api_docs/python/tf/keras/models) object. Consider again a neural network with just one hidden layer:"
      ]
    },
    {
      "metadata": {
        "id": "ieyztj5VvAGr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "class SingleHiddenLayerNetwork(tf.keras.Model):\n",
        "  def __init__(self):\n",
        "    super(SingleHiddenLayerNetwork, self).__init__()\n",
        "    self.hidden_layer = tf.layers.Dense(10, activation=tf.nn.tanh, use_bias=True)\n",
        "    self.output_layer = tf.layers.Dense(3, use_bias=True, activation=None)\n",
        "\n",
        "  def call(self, x):\n",
        "    return self.output_layer(self.hidden_layer(x))\n",
        "\n",
        "net = SingleHiddenLayerNetwork()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CxIseRSnvsAm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "It is relatively similar to the [functional interface of Keras](https://keras.io/getting-started/functional-api-guide/). If you are working with v1.5 or v1.6, you need to use `tfe.Network` instead of `tf.keras.Model`. Note that variables are not yet initialized:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "fk6OE_IJvqJL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "430f7ac9-fdf0-4620-c0e5-3cdd4fa925ab",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043343286,
          "user_tz": -60,
          "elapsed": 487,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "len(net.variables)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "metadata": {
        "id": "4mF3xz9DwWYn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "However, it is enough to use the model a single time to automatically trigger the initialization. Networks objects can be called as if they were functions:"
      ]
    },
    {
      "metadata": {
        "id": "kxHXkU-awT57",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "634d2156-53ef-4492-b7f4-1ad8b1bda391",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043345647,
          "user_tz": -60,
          "elapsed": 523,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net(tf.constant(X_train[0:1]))\n",
        "len(net.variables)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "metadata": {
        "id": "fkQcTsWfmD9G",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Networks have several additional utilities to handle the models. For example, we can count the number of adaptable parameters of the model:"
      ]
    },
    {
      "metadata": {
        "id": "S3cH5OzDl_wv",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4a16a374-387b-41a0-940c-fdf37228e864",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043347527,
          "user_tz": -60,
          "elapsed": 502,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net.count_params()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "83"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "metadata": {
        "id": "BpqJR0Urmpd5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For simple sequential models, eager execution also provides a short-hand with the `Sequential` object; the following is equivalent to the previous model definition:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VuQoVc_cepvV",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = tfe.Sequential(layers_funcs=[\n",
        "    tf.layers.Dense(10, activation=tf.nn.tanh, use_bias=True), \n",
        "    tf.layers.Dense(3, use_bias=True, activation=None)\n",
        "])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kfB5YWC2xnus",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Handling the input pipeline"
      ]
    },
    {
      "metadata": {
        "id": "CUvEWaFk2FZ3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The next step is to define the input function to handle our dataset. Since the latest versions of TF, the preferred method is the use of the [data pipeline with the Dataset object](https://www.tensorflow.org/programmers_guide/datasets):"
      ]
    },
    {
      "metadata": {
        "id": "uUhI33Ugwc1t",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "train_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SORXDHDL2izp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To cycle over the data, we can use the eager module `tfe.Iterator` instead of `tf.data.Iterator`, which acts as a standard Python iterator. For example, we can cycle over each batch in the dataset and print the proportion of the first class in each batch:\n"
      ]
    },
    {
      "metadata": {
        "id": "Q2MeCbe-2h0o",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "464c3c5a-bf60-43bd-be58-3d8b1c4329b3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043379635,
          "user_tz": -60,
          "elapsed": 479,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "for xb,yb in tfe.Iterator(train_dataset.batch(32)):\n",
        "  print('Percentage of class [0]: ', tf.reduce_mean(yb[:, 0]).numpy()*100, ' %')"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Percentage of class [0]:  40.625  %\n",
            "Percentage of class [0]:  34.375  %\n",
            "Percentage of class [0]:  21.875  %\n",
            "Percentage of class [0]:  37.5  %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VJO-AXD5SpXp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "`from_tensor_slices` creates a dataset having one element for each row of our original tensors. If you don't need batching, `from_tensors` will treat the entire tensor as a single element:"
      ]
    },
    {
      "metadata": {
        "id": "cRDP35hUS7eR",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "40cac023-d8fe-467f-e5d2-98519b988e97",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043381942,
          "user_tz": -60,
          "elapsed": 528,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "train_dataset_alt = tf.data.Dataset.from_tensors((X_train, y_train))\n",
        "for xb, yb in tfe.Iterator(train_dataset_alt):\n",
        "  # Check that the batch is equivalent to the entire training array\n",
        "  assert(np.all(X_train == xb.numpy()))\n",
        "  # Compute the percentage of labels for the first class\n",
        "  print('Percentage of class [0] (entire dataset): ', tf.reduce_mean(yb[:, 0]).numpy()*100, ' %')"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Percentage of class [0] (entire dataset):  33.035714285714285  %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "glIRAB6PHUIl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can apply further transformations to the dataset before processing it, such as repeating the entire dataset twice:"
      ]
    },
    {
      "metadata": {
        "id": "MA4DNEYpHTeG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "1cdef2a2-ba36-485b-b6cc-f3a0930f1d0e",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043383216,
          "user_tz": -60,
          "elapsed": 487,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "for xb,yb in tfe.Iterator(train_dataset.repeat(2).batch(32)):\n",
        "  print('Percentage of class [0]: ', tf.reduce_mean(yb[:, 0]).numpy()*100, ' %')"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Percentage of class [0]:  40.625  %\n",
            "Percentage of class [0]:  34.375  %\n",
            "Percentage of class [0]:  21.875  %\n",
            "Percentage of class [0]:  43.75  %\n",
            "Percentage of class [0]:  28.125  %\n",
            "Percentage of class [0]:  34.375  %\n",
            "Percentage of class [0]:  28.125  %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "vt3DD2EvHsuH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Or shuffling the dataset each time we cycle over it:"
      ]
    },
    {
      "metadata": {
        "id": "gEVoay9KHvcL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "e1f0fee1-dfab-46ac-d901-43e8c0f06bcd",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043385310,
          "user_tz": -60,
          "elapsed": 529,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "for xb,yb in tfe.Iterator(train_dataset.shuffle(1000).batch(32)):\n",
        "  print('Percentage of class [0]: ', tf.reduce_mean(yb[:, 0]).numpy()*100, ' %')"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Percentage of class [0]:  31.25  %\n",
            "Percentage of class [0]:  34.375  %\n",
            "Percentage of class [0]:  37.5  %\n",
            "Percentage of class [0]:  25.0  %\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pV6WsMHpH1PX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The parameter for the `shuffle` method is a buffer dimension: if we need to process very large datasets, it would be unfeasible to shuffle them uniformly. In this case, the buffer size specifies the dimension of blocks that we will randomly shuffle. If we set the buffer parameter larger than the size of the dataset, this is equivalent to a uniform shuffling.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ehLIBkN23ltL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Gradients' computation and model's optimization"
      ]
    },
    {
      "metadata": {
        "id": "0oiZAEHc3qQg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "It is now time to define our cost function:"
      ]
    },
    {
      "metadata": {
        "id": "25ReOH_e3aSf",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def loss(net, inputs, labels):\n",
        "  return tf.reduce_sum(tf.nn.softmax_cross_entropy_with_logits_v2(\n",
        "          logits=net(inputs), labels=labels))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oHy0tkywJDSj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We are using `softmax_cross_entropy_with_logits_v2` instead of the older `softmax_cross_entropy_with_logits`, which would throw a warning as it will be deprecated in future releases. The two are identical, except that the new version allows to back-propagate through the labels by default: while this is entirely useless in this case, it is needed when the labels are coming from another neural model (as in [generative networks](https://www.tensorflow.org/api_docs/python/tf/contrib/gan)).\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "-3q6s47i5k58",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note again that the cost is defined explicitly as a Python function, instead of as a given node in a TensorFlow graph."
      ]
    },
    {
      "metadata": {
        "id": "AQ8zKQfw53xX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In order to better understand the eager execution, we are going to explicitly compute the gradients and apply them via the `apply_gradients` function of the optimizer. In particular, `tfe.implicit_gradients` works like `tfe.gradients_function`, but automatically returns the gradients of all variables of our model:"
      ]
    },
    {
      "metadata": {
        "id": "HflpnsYI6ryG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "loss_grad = tfe.implicit_gradients(loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4ff-YHPhInX_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Once again, another variant is the `implicit_value_and_gradients`, which returns both the value of the function and its gradients:"
      ]
    },
    {
      "metadata": {
        "id": "JJRXGWCLIvTw",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "loss_and_grads = tfe.implicit_value_and_gradients(loss)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "J5shPzgc7Gam",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Using the first syntax, the optimization cycle is now a trivial matter:"
      ]
    },
    {
      "metadata": {
        "id": "jojDeSP462_W",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 3
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "df2a1144-b1dc-4d2f-a94b-630b2527eb58",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043434079,
          "user_tz": -60,
          "elapsed": 1258,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = SingleHiddenLayerNetwork()\n",
        "opt = tf.train.AdamOptimizer(learning_rate=0.01)\n",
        "\n",
        "# Loop over the epochs\n",
        "for epoch in range(50):\n",
        "  \n",
        "  # For each epoch we shuffle the dataset\n",
        "  for (xb, yb) in tfe.Iterator(train_dataset.shuffle(1000).batch(32)):\n",
        "    opt.apply_gradients(loss_grad(net, xb, yb))\n",
        "    \n",
        "  # Training accuracy at the end of each tenth epoch\n",
        "  if epoch % 10 == 0:\n",
        "    print(\"Epoch %d: Loss on training set : %f\" %\n",
        "              (epoch, loss(net, X_train, y_train).numpy()))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0: Loss on training set : 125.593775\n",
            "Epoch 10: Loss on training set : 38.101423\n",
            "Epoch 20: Loss on training set : 20.658543\n",
            "Epoch 30: Loss on training set : 12.575747\n",
            "Epoch 40: Loss on training set : 8.507938\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "NJ44xWB59E5P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note how, when compared to the classical TensorFlow low-level interface, the previous code tends to be more readable, and closely resembles what the code would have looked like had we only used NumPy."
      ]
    },
    {
      "metadata": {
        "id": "RBeBo-BEpZq4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the latest release, the inner optimization can also be simplified using the standard `minimize` interface of the optimizer, by constructing a parameter-free anonymous function:\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "cDnBCnDqpz1u",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "opt.minimize(lambda: loss(net, xb, yb))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XecSJK4r9PD1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluating the model and plotting the results"
      ]
    },
    {
      "metadata": {
        "id": "5ix3nr8wqglP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Computing and visualizing metrics"
      ]
    },
    {
      "metadata": {
        "id": "GFVome_Y9TBj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Two common debug tools in TensorFlow are the **summaries** and the **metrics**, which have a corresponding new implementation suitable to the eager execution. Starting from the latter, a set of pre-defined metrics can be found in the `tfe.metrics` module, which can be used in a standard object-oriented fashion:"
      ]
    },
    {
      "metadata": {
        "id": "K6NXRnzL8JG5",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy = tfe.metrics.Accuracy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Wfo4i218-B7P",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can accumulate values inside the metric and print an average result as follows:"
      ]
    },
    {
      "metadata": {
        "id": "nQXhKqrJ8Yhq",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7520e0b2-4ee8-413b-f304-e523dffbf4ea",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043440131,
          "user_tz": -60,
          "elapsed": 508,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "accuracy(tf.argmax(net(tf.constant(X_test)), axis=1), tf.argmax(tf.constant(y_test), axis=1))\n",
        "print('Final test accuracy is: ', accuracy.result().numpy())"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Final test accuracy is:  0.9210526315789473\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_n7Gx2FkZT_v",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let us rewrite the optimization code, this time by accumulating the training accuracy at each epoch:"
      ]
    },
    {
      "metadata": {
        "id": "_mokLkh-YYW9",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = SingleHiddenLayerNetwork()\n",
        "opt = tf.train.AdamOptimizer(learning_rate=0.01)\n",
        "\n",
        "# Numpy array to keep track of the accuracy\n",
        "acc_history = np.zeros(50)\n",
        "\n",
        "# Loop over the epochs\n",
        "for epoch in range(50):\n",
        "  \n",
        "  # Initialize the metric\n",
        "  accuracy = tfe.metrics.Accuracy()\n",
        "  \n",
        "  # For each epoch we shuffle the dataset\n",
        "  for (xb, yb) in tfe.Iterator(train_dataset.shuffle(1000).batch(32)):\n",
        "    opt.apply_gradients(loss_grad(net, xb, yb))\n",
        "    \n",
        "    # Save the training accuracy on the batch\n",
        "    accuracy(tf.argmax(net(tf.constant(xb)), axis=1), tf.argmax(tf.constant(yb), axis=1))\n",
        "  \n",
        "  # Save the overall accuracy in our vector\n",
        "  acc_history[epoch] = accuracy.result().numpy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7IWWXM6Wcn-f",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can use Matplotlib to plot the resulting accuracy:"
      ]
    },
    {
      "metadata": {
        "id": "d4bmyy8TaURr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 361
        },
        "outputId": "b4ee40b4-5617-41df-b63b-7f2a81191e64",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043444262,
          "user_tz": -60,
          "elapsed": 543,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.figure()\n",
        "plt.plot(acc_history)\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.show()"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAFYCAYAAAB6RnQAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt4VOW59/HfJJMQkgxJJsyE8ymc\ng0FRopiKikAt2oNWNNSKtWq3pe7WVjylZcddN4jWvTfWq2+3r6drlyoNcmh9rTYWhYoaOVkDRBSI\nggGRzOQccp6Z9w9gBCWEhFmzZs18P39lzUxm7rkb+XU961nPYwsEAgEBAADLiDO7AAAA0DOENwAA\nFkN4AwBgMYQ3AAAWQ3gDAGAxhDcAABZjN7uAM+XxNIb0/TIyklVb2xzS94xV9DJ06GXo0MvQoZeh\n09NeulyOUz4es2fednu82SVEDXoZOvQydOhl6NDL0AlVL2M2vAEAsCrCGwAAiyG8AQCwGEPDe/fu\n3Zo5c6b++Mc/fuW5d955R9ddd51uuOEG/e53vzOyDAAAooph4d3c3KyHHnpI06ZNO+Xz//Ef/6En\nnnhCK1as0Ntvv629e/caVQoAAFHFsPBOTEzUU089Jbfb/ZXnKisrlZaWpoEDByouLk6XXnqpSktL\njSoFAICoYlh42+12JSUlnfI5j8cjp9MZPHY6nfJ4PEaVAgBAVLHMIi0ZGckhv9ewq5vf0XP0MnTo\nZejQy9Chl6ETil6aEt5ut1terzd4fPjw4VMOr58o1Kv7uFyOkK/aFqvoZejQy9Chl6FDL0Onp72M\nqBXWhgwZoqamJh04cECdnZ1av3698vPzzSgFAADLMezMe+fOnXrkkUd08OBB2e12lZSUaMaMGRoy\nZIhmzZqlBx98UHfffbckac6cORo5cqRRpQBASHX6/Nq867DqmtrNLiUsUlL66MiRNrPLiHjpqYma\nljNANpvN8M8yLLwnTZqk5cuXd/n81KlTVVxcbNTHA0DIBQIBle2tVvH6vTpcw0Yd+KpJozLVLznR\n8M+xzIQ1ADBTZVWT/vT6Hu3aX6s4m02XTxmsydmZkow/yzJbWlpf1de3mF1GxEtPTQxLcEuENwCc\nVv2Rdq1982Nt3P6ZAgFp0iinbpgxRoP7p5hdWtgwYS3yEN4AcAodnT69tqVSfy3dr9Z2nwb1T9EN\nM0brnFGZZpcGEN4Aoou3rkV/efsTeRva1Nnh6/X7VDe0qq6pXal9E/T92dm69NxBio9jLydEBsIb\nQFRoaevUX0v367Utler0+RUfZ9PZTPpNsMfpyrxhuvri4UpOSghdoUAIEN4ALM3vD2jj9s+09s2P\n1dDcoQxHH113Wbaunj5a1dVNZpcHGILwBmBZ5ftqVPz6Hh3wHFFiQpy+c8lIfT1vmPokxCsuLvpn\ngSN2Ed4AIk5jc7v8/kCXzx+fAV5WUS2bpPxzBuja6dnKcPQJX5GAiQhvABHjgKdJxa/vUfm+2jN6\n/bih6Sq4YoyGD2DTDMQWwhuA6RqOtOvPGz/WP8qO3ks9ekianKc5i46z2XT+OLemjO0flqUogUhD\neAMwTUenT+u2HtDLpfvU0ubTwMzk4L3UhDLQNcIbQNgFAgFt/cijF9fvlbe+Val9E3TjrKP3Utvj\nuZca6A7hDVhEa3untn3kUXun3+xSzkogENCmDw5rz4F6xcfZNHvqUH0zf4RSuJcaOGOEN2ABHZ0+\nLXtxu3ZX1pldSsicN6a/rr98tLKcyWaXAlgO4Q1EOH8goKdf3qXdlXU6b0x/5U3IMruks+bO6KuR\nA/uZXQZgWYQ3EOFWvrFXWz6s0tghabrj2zlKsMebXRIAkzEzBIhgr22p1GtbKjUwM1l3fjeX4AYg\nifAGItbWD6tU/PoepaUm6ufXT1ZqXyZ0ATiK8AYi0O7KOv3f//eBEhPj9fO5k9U/ra/ZJQGIIIQ3\nEGE+8x7RE6u3KxAI6CfXTNKwLJb+BHAywhuIIDUNrfrvlWU60tqpH3xjvCaNzDS7JAARiNnmwBnq\n6PQp0PVGV2etvdOvZcu3qbqhVddMH6X8cwYa92EALI3wBrpxwNOk4jf2qvyTmrB83qXnDtLV04aH\n5bMAWBPhDXThyztdjRjgUGqysTO+xw136sqpQ9iUA8BpEd7Al3R0+rVua6UpO125XA55PI2GfgYA\n6yO8gWMCgYC2feTRymM7XaUk2XXjrLHsdAUg4hDeiAl7DtSpuqG1y+f9/oDefP8z7WanKwAWQHgj\n6q3bWqkX1u05o9ey0xUAKyC8EdW2fVSlFev2qF9Kor6dP+K016yHuFI1ekhaGKsDgN4hvBG19hw4\ntsRowtElRocPYKUyANGBWTiISoeqj+i3q7bL5wtowTWTCG4AUYXwRtSpb2oLLjF685XjdM4olhgF\nEF0Ib0SV1vZOLXtxu7z1rfr210bqksmDzC4JAEKO8EbU6PT59X/+vFP7DzfqktyB+lb+CLNLAgBD\nEN6ICoFAQH8o+Ug7P67ROaMyddPXx7HEKICoRXgjKrz09j69tf2Qhg9w6MffyWFFNABRjVvFENE+\nPdyoF9fv1Yef1p32dT5/QP3TknTX3MlKSuTPGkB04185RKT6pjatefNjvbX9kAI6uoBKn8Suz6ZT\nkhJUcMUYpaUkhq9IADAJ4Y2I0t7h02tbKvXXd/errd2nwf1TdMMVozVpJLd7AcBxhDciQiAQ0OZd\nVVq1Ya+qG9rkSE7Q9ZeP1vTJAxUfx/VrADgR4Q1D+QMB7amsU1uHr8vXtHf4VbLlU1UcbJA93qYr\nLxymq6eNUHISf54AcCr86wjDfPRprf70+l7tP9x4Rq8/f5xLcy8fLXd6X4MrAwBrI7wRclW1zXpx\nfYW27fZIkvImuDU86/Rri48Zks6OXgBwhghvhExza6defmef1m2rVKcvoOzB/VRwxRhlDyKUASCU\nDA3vJUuWqKysTDabTYWFhcrNzQ0+t27dOv3+979XYmKirrrqKn3/+983shQYyOfza/17B7R24ydq\naulQZr8kzb08W1PHu1nlDAAMYFh4b968Wfv371dxcbEqKipUWFio4uJiSZLf79dDDz2ktWvXKj09\nXbfffrtmzpypAQMGGFUOemnrh1Xa+UnNaV+z73CjPv28UX0S4/XdS0dp1gVDlZgQH6YKASD2GBbe\npaWlmjlzpiQpOztb9fX1ampqUmpqqmpra9WvXz85nU5J0kUXXaR33nlH1157rVHloBeaWjr09Msf\nqL3Tf9rX2WzS9MkDdc0lo5SW2idM1QFA7DIsvL1er3JycoLHTqdTHo9HqampcjqdOnLkiPbt26fB\ngwdr06ZNysvLO+37ZWQky24P7dmcy3X6SVSxbsPru9Xe6df3Zo/T9ClDunxdcpJdGY6kMFYW3fi7\nDB16GTr0MnRC0cuwTVgLBALBn202m5YuXarCwkI5HA4NGdJ1MBxXW9sc0npcLoc8njO7hSkWdfr8\neunNCiUlxis/J0uJCnT52gxHEr0MEf4uQ4dehg69DJ2e9rKroDds6Sq32y2v1xs8rqqqksvlCh7n\n5eXphRde0JNPPimHw6HBgwcbVQp6YcuHVapratcluYPUtw83JQBAJDEsvPPz81VSUiJJKi8vl9vt\nVmpqavD52267TdXV1Wpubtb69es1bdo0o0pBDwUCAb22uVI2mzTzgu5HRQAA4WXYKdWUKVOUk5Oj\ngoIC2Ww2FRUVac2aNXI4HJo1a5auv/56/fCHP5TNZtOPfvSj4OQ1mG93ZZ32H27U+WNdcrHaGQBE\nHEPHQxcuXHjS8fjx44M/z549W7Nnzzby49FLr22plCTNmjrU5EoAAKfCdk04SVVts97f49WIAQ6N\nYblSAIhIhDdOsm7rAQUkzc4byupoABChCG8ENbd2auOOQ8pw9NEF49xmlwMA6ALhjaA3yz5TW7tP\nV5w/RPZ4/jQAIFLxLzQkST6/X69vq1RiQpwuPXeQ2eUAAE6D8IYkadtHHlU3tCn/nIFKSUowuxwA\nwGkQ3pAk/f347WEXcHsYAEQ6whuqOFivis8aNDk7UwOcyWaXAwDoBuGN4KIss1mUBQAsgfCOcdX1\nrdr2kUdDXKkaPzzD7HIAAGeA8I5xr287IH8goNlTWZQFAKyCvR5jVCAQ0I6Pa/SPsoPql5KoCydm\nmV0SAOAMEd4x6ICnScVv7FX5JzWy2aQbZoxRgp1BGACwCsI7hjQ0t+vPGz/RP94/qEBAyhmRoRtm\njNEQd2r3vwwAiBiEdwzo6PRr3bZKvfzOPrW0+TTAmawbZoxWbnYm17kBwIII7ygWCAS07SOPVq7f\nK299q1KS7Lpx1lhdeu4g1i4HAAsjvKPYG+8d1PN/3634OJtmTx2qb+aPYOlTAIgChHeUajjSrjVv\nfqzkPnb9cv75GpiZYnZJAIAQYew0Sq36R4Va2jp1zfRRBDcARBnCOwp9/FmD3tp+SENcKbrsPLb3\nBIBoQ3hHGX8goOf/vluSdOOssYqP439iAIg2/MseZd7ecUifHGpQ3gS3xg1jrXIAiEaEdxRpbu3U\n6g0VSkyI0/WXjza7HACAQQjvKPKXtz5RQ3OHrp42Qs5+SWaXAwAwCOEdJQ56mvT6tgNyp/fV1/PY\nlxsAohnhHQUCgYBeWLdH/kBABVeMUYI93uySAAAGIryjwLaPPNq1v1bnjMrU5NGZZpcDADAY4W1x\nbR0+Fb+xR/FxNs2bOYaNRgAgBhDeFvfqu/tV3dCm2XlDNcCZbHY5AIAwILwtzFPXolfe/VTpqYn6\n5sUjzC4HABAmhLdFdfr8eu6VXer0+TX38tFKSmSPGQCIFYS3BQUCAT33yi59+GmdzhvTXxdNzDK7\nJABAGBHeFrTmzY9VWn5Y2YP66UffymGSGgDEGMLbYta/d0B/Ld2vrIy++ul1ueqTwD3dABBrCG8L\n+ecej/74993ql5ygn18/WY7kRLNLAgCYgPC2iIrP6vXkX8qVYI/Tz+ZOljuD28IAIFYR3hZwuKZZ\nj7+4XR0+v3787UkaObCf2SUBAExEeEe4hiPt+u+VZWpq6dBNXx+nyaP7m10SAMBkhHcEa2v36fFV\nZaqqa9HVF4/QZecONrskAEAEYGUPkzQ0t+sPf/tIza0dXb6mrqldn9c0K3/SAF1zycgwVgcAiGSE\nt0lWra/Qe7s93b7u/HEu3fyN8dzLDQAIIrxNUPFZvd7acUhDXKn6tx9coLi4roM5jtAGAHwJ4R1m\n/kBAL/x9tyTpxlljZI9n2gEAoGcMDe8lS5aorKxMNptNhYWFys3NDT73/PPP66WXXlJcXJwmTZqk\nX/7yl0aWEjHe3n5Inxxq1IUTszRuWIbZ5QAALMiw077Nmzdr//79Ki4u1uLFi7V48eLgc01NTXrm\nmWf0/PPPa8WKFaqoqND7779vVCkRo7m1Q6v+UaHEhDjNvSzb7HIAABZlWHiXlpZq5syZkqTs7GzV\n19erqalJkpSQkKCEhAQ1Nzers7NTLS0tSktLM6qUiPHntz5RY3OHvnnxCDn7JZldDgDAogwLb6/X\nq4yML4aFnU6nPJ6js6v79Omjn/zkJ5o5c6Yuv/xyTZ48WSNHRvetUAc8TXpj20G5M/pq9tRhZpcD\nALCwsE1YCwQCwZ+bmpr05JNP6m9/+5tSU1N1880368MPP9T48eO7/P2MjGTZ7aHdQcvlcoT0/boS\nCAS0bNV2+QMB3XFtrgYNjL5RhnD1MhbQy9Chl6FDL0MnFL00LLzdbre8Xm/wuKqqSi6XS5JUUVGh\noUOHyul0SpIuuOAC7dy587ThXVvbHNL6XC6HPJ7GkL5nV7Z+WKXte73Kzc7UCFdK2D43XMLZy2hH\nL0OHXoYOvQydnvayq6A3bNg8Pz9fJSUlkqTy8nK53W6lpqZKkgYPHqyKigq1trZKknbu3KkRI0YY\nVYqp2jp8Kn5jj+zxNs27YozZ5QAAooBhZ95TpkxRTk6OCgoKZLPZVFRUpDVr1sjhcGjWrFm69dZb\nNX/+fMXHx+u8887TBRdcYFQppnqldL+qG9o056LhynKyjScA4OwZes174cKFJx2fOCxeUFCggoIC\nIz/edFV1LXp106dKT03U1RcPN7scAECUYHkvAxW/vkedPr+uv3y0khJZzA4AEBqEt0HK99Xon3u8\nGjskTRdOzDK7HABAFCG8DfLPYzuGXXtpNjuCAQBCivA2SG1jmyRpUP8UkysBAEQbwtsgNQ1tSrTH\nKSWJa90AgNAivA1S29iqDEcfhswBACFHeBugo9OvhuYOZTj6mF0KACAKEd4GqGs6er07w8HOYQCA\n0CO8DVDTcHTZV2c/zrwBAKHXbXhXVFSEo46ocnymuZNhcwCAAboN75/+9KeaN2+eVq9erZaWlnDU\nZHnHw5thcwCAEbq9j+mvf/2rdu/erVdffVU33XSTJkyYoLlz5yo3Nzcc9VlSTcOxM2+GzQEABjij\na95jx47Vz372M91///2qqKjQggULdOONN2rfvn0Gl2dNNY1Hr3kz2xwAYIRuz7wPHjyotWvX6uWX\nX9bo0aN1xx136JJLLtGOHTt0zz336MUXXwxHnZZS29gme3ycUvsmmF0KACAKdRveN910k6677jr9\n7//+r7KyvthgIzc3l6HzLtQ2tsnJAi0AAIN0O2z+0ksvacSIEcHgXrFihY4cOSJJWrRokbHVWVCn\nz6+GI+1c7wYAGKbb8H7ggQfk9XqDx62trbr33nsNLcrK6hrbFBDXuwEAxuk2vOvq6jR//vzg8S23\n3KKGhgZDi7KyGm4TAwAYrNvw7ujoOGmhlp07d6qjo8PQoqyMmeYAAKN1O2HtgQce0IIFC9TY2Cif\nzyen06lHH300HLVZUnB1Na55AwAM0m14T548WSUlJaqtrZXNZlN6erree++9cNRmSbXHF2hh2BwA\nYJBuw7upqUl/+ctfVFtbK+noMPrq1av11ltvGV6cFX2xNCpn3gAAY3R7zfuuu+7SRx99pDVr1ujI\nkSNav369HnzwwTCUZk01ja2yx9vkSGaBFgCAMboN77a2Nv3617/W4MGDdd999+kPf/iDXn311XDU\nZkk1jW3KYIEWAICBzmi2eXNzs/x+v2pra5Wenq7Kyspw1GY5nT6/GprauU0MAGCobq95f/vb39bK\nlSs1d+5czZkzR06nU8OHDw9HbZZT13R0gRb28QYAGKnb8C4oKAgOAU+bNk3V1dWaMGGC4YVZUXCy\nGreJAQAM1O2w+Ymrq2VlZWnixIlcz+1C8B5vhs0BAAbq9sx7woQJevzxx3XeeecpIeGLGdTTpk0z\ntDArqmngNjEAgPG6De9du3ZJkrZu3Rp8zGazEd6ncHxpVFZXAwAYqdvwXr58eTjqiAq1bEoCAAiD\nbsP7e9/73imvcT///POGFGRltY1tio9jgRYAgLG6De+77ror+HNHR4feffddJScnG1qUVdU0tCrD\n0UdxTOgDABio2/DOy8s76Tg/P1+33367YQVZVafPr/qmdo0ZkmZ2KQCAKNdteH95NbVDhw7pk08+\nMawgq2o40q6ApIx+XO8GABir2/C++eabgz/bbDalpqbqzjvvNLQoK6phNzEAQJh0G95vvPGG/H6/\n4uKOrufS0dFx0v3eOKqm4ehtYoQ3AMBo3a6wVlJSogULFgSPb7zxRv3tb38ztCgrYnU1AEC4dBve\nzz33nH7zm98Ej5999lk999xzhhZlRcHwZoEWAIDBug3vQCAgh8MRPE5NTWVt81Ng2BwAEC7dXvOe\nNGmS7rrrLuXl5SkQCGjjxo2aNGlSOGqzlOMLtPRLSTS7FABAlOs2vH/1q1/ppZde0vbt22Wz2fSt\nb31LV155ZThqs5Saxjalp7JACwDAeN2Gd0tLixISErRo0SJJ0ooVK9TS0qKUlBTDi7MKn//oAi2j\nBvczuxQAQAzo9pr3fffdJ6/XGzxubW3Vvffea2hRVlPf1C5/ICAn17sBAGHQ7Zl3XV2d5s+fHzy+\n5ZZb9MYbb5zRmy9ZskRlZWWy2WwqLCxUbm6uJOnw4cNauHBh8HWVlZW6++679c1vfrOn9UcEbhMD\nAIRTt+Hd0dGhiooKZWdnS5J27Nihjo6Obt948+bN2r9/v4qLi1VRUaHCwkIVFxdLkrKysoJbjXZ2\nduqmm27SjBkzzuZ7mKqW1dUAAGHUbXg/8MADWrBggRobG+X3+5WRkaFHH3202zcuLS3VzJkzJUnZ\n2dmqr69XU1OTUlNTT3rd2rVr9fWvf93S19C5TQwAEE7dhvfkyZNVUlKiQ4cOadOmTVq7dq1+/OMf\n66233jrt73m9XuXk5ASPnU6nPB7PV8L7xRdf1LPPPtttoRkZybLb47t9XU+4XI7uX3QGWn0BSVL2\ncGfI3tNqYvV7G4Fehg69DB16GTqh6GW34f3+++9rzZo1euWVV+T3+/XQQw9p9uzZPf6gQCDwlcf+\n+c9/atSoUV8J9FOprW3u8WeejsvlkMfTGJL3Onj42Pt0+kL2nlYSyl7GOnoZOvQydOhl6PS0l10F\nfZezzZ966inNmTNHP//5z+V0OrV69WoNGzZMV1111RltTOJ2u0+apV5VVSWXy3XSazZs2KBp06ad\n6XeIWLWNbYqz2ZTGAi0AgDDoMryXLVumhIQEPfzww7rrrrs0fPjwHi2Lmp+fr5KSEklSeXm53G73\nV86wd+zYofHjx/ey9MhR09iqdEei4uJYoAUAYLwuh803bNigtWvXqqioSH6/X9dcc80ZzTI/bsqU\nKcrJyVFBQYFsNpuKioq0Zs0aORwOzZo1S5Lk8XiUmZl59t/CRH5/QHWN7Ro1iAVaAADhYQuc6mL0\nl2zZskWrV69WSUmJLrzwQs2bN0+XXnppOOoLCvX1llBdw6ltbNPdv3tbU8e79ePvxOaa71wPCx16\nGTr0MnToZegYfs37RFOnTtXSpUu1ceNGXXbZZfrd7353xh8c7WoauU0MABBeZxTex6WmpqqgoEAr\nV640qh7LqW04vroa4Q0ACI8ehTe+Krg0aj+WRgUAhAfhfZZYGhUAEG6E91nimjcAINwI77NUc2yB\nlvRUwhsAEB6E91mqbWhTWioLtAAAwofwPgt+f0B1TW3MNAcAhBXhfRYamtvl8we43g0ACCvC+yxw\nmxgAwAyE91moaeA2MQBA+BHeZ4HbxAAAZiC8zwLD5gAAMxDeZyEY3px5AwDCiPA+CzUNrbLZpH4p\niWaXAgCIIYT3WahtbFNaSqLs8bQRABA+pE4v+QMB1Ta2cb0bABB2hHcvNTZ3sEALAMAUhHcv1TRw\nmxgAwByEdy99MdOcYXMAQHgR3r30xT3enHkDAMKL8O4lhs0BAGYhvHvp+Jk34Q0ACDfCu5dqGttk\nk5SeSngDAMKL8O6Ftnaf9n/eKHdGXxZoAQCEHcnTC//c61Fbh09TJ2SZXQoAIAYR3r2wqfywJOmi\niYQ3ACD8CO8eamrp0M5PajQsK1WD+qeYXQ4AIAYR3j205cMq+fwBXTRxgNmlAABiFOHdQ5vKP5dN\nUt4Et9mlAABiFOHdA976Fu0+UK9xw9LZTQwAYBrCuwc276qSJF2Uw5A5AMA8hHcPvFv+uezxNp0/\nzmV2KQCAGEZ4n6EDVU064Dmic0ZlKiUpwexyAAAxjPA+Q+9+cOzebobMAQAmI7zPgD8Q0KYPDisp\nMV6TszPNLgcAEOMI7zOw90C9qhtadf5YlxIT4s0uBwAQ4wjvM7CJIXMAQAQhvLvR6fNry4dV6peS\nqPHD080uBwAAwrs75Z/UqKmlQ3kT3IqPo10AAPORRt0IDpmzljkAIEIQ3qfR2t6p9/Z45M7oq5ED\nHWaXAwCAJML7tN7f41V7h18XTcySzWYzuxwAACRJdiPffMmSJSorK5PNZlNhYaFyc3ODzx06dEi/\n+MUv1NHRoYkTJ+rXv/61kaX0yvGFWS6cmGVyJQAAfMGwM+/Nmzdr//79Ki4u1uLFi7V48eKTnl+6\ndKl++MMfatWqVYqPj9dnn31mVCm90tDcrp0f12j4AIcGZqaYXQ4AAEGGhXdpaalmzpwpScrOzlZ9\nfb2ampokSX6/X9u2bdOMGTMkSUVFRRo0aJBRpfTK1g+r5A8ENI2zbgBAhDEsvL1erzIyMoLHTqdT\nHo9HklRTU6OUlBQ9/PDDmjdvnv7zP//TqDJ67d0PDssmaeoEwhsAEFkMveZ9okAgcNLPhw8f1vz5\n8zV48GD96Ec/0oYNG3TZZZd1+fsZGcmy20O7NKnLdeoZ5PVNbdp7oF65o/tr7Kj+If3MaNVVL9Fz\n9DJ06GXo0MvQCUUvDQtvt9str9cbPK6qqpLLdXQf7IyMDA0aNEjDhg2TJE2bNk179uw5bXjX1jaH\ntD6XyyGPp/GUz+09WC9JGpSZ3OVr8IXT9RI9Qy9Dh16GDr0MnZ72squgN2zYPD8/XyUlJZKk8vJy\nud1upaamSpLsdruGDh2qffv2BZ8fOXKkUaX0mLeuRZLUPy3J5EoAAPgqw868p0yZopycHBUUFMhm\ns6moqEhr1qyRw+HQrFmzVFhYqPvvv1+BQEBjx44NTl6LBN76VkmENwAgMhl6zXvhwoUnHY8fPz74\n8/Dhw7VixQojP77XvgjvviZXAgDAV7HC2il4648Om2dy5g0AiECE9yl461vVLzlBfRJCO7sdAIBQ\nILy/xB8IqLq+Vf3TGTIHAEQmwvtL6hrb5PMHmKwGAIhYhPeXHJ+sxvVuAECkIry/pPpYeLuYaQ4A\niFCE95d46lmgBQAQ2QjvL2HYHAAQ6QjvL6lmdTUAQIQjvL/EU9eitNREJYR4BzMAAEKF8D6Bz+9X\nbWMbZ90AgIhGeJ+grrFdPn+AmeYAgIhGeJ+ANc0BAFZAeJ+ArUABAFZAeJ+ArUABAFZAeJ/g+LB5\n/3TOvAEAkYvwPoG3rlU2SU4H4Q0AiFyE9wm89a1Kd/RRgp22AAAiFyl1DPd4AwCsgvA+pqahTf4A\n+3gDACIf4X3MFxuSMNMcABDZCO9jjs80d3HmDQCIcIT3Md46FmgBAFgD4X1McNg8nWFzAEBkI7yP\nqa5vkc0mOR19zC4FAIDTIryP8dS3yunoI3s8LQEARDaSSlKnz6+6xjZmmgMALIHwllTT0KqAmGkO\nALAGwltHh8wl9vEGAFgD4S1VQHbTAAAKGUlEQVSpmq1AAQAWQnjrhK1AOfMGAFgA4a0TFmhhH28A\ngAUQ3jq6QEuczaYM7vEGAFgA4a2jw+bOfn0UH0c7AACRL+bTqqPTp7qmdq53AwAsI+bDu7qhTRIz\nzQEA1hHz4c1McwCA1RDezDQHAFgM4c0CLQAAiyG8GTYHAFgM4V3fqvg4m9JTuccbAGANhHd9qzL7\nJSkuzmZ2KQAAnJGYDu/2Dp8ajrSzmxgAwFJiOryPT1ZzMdMcAGAhdiPffMmSJSorK5PNZlNhYaFy\nc3ODz82YMUMDBgxQfHy8JOmxxx5TVlaWkeV8hTe4jzczzQEA1mFYeG/evFn79+9XcXGxKioqVFhY\nqOLi4pNe89RTTyklJcWoErpVzUxzAIAFGTZsXlpaqpkzZ0qSsrOzVV9fr6amJqM+rlc8wXu8CW8A\ngHUYdubt9XqVk5MTPHY6nfJ4PEpNTQ0+VlRUpIMHD+r888/X3XffLZut6xnfGRnJstvjQ1pjY2un\nJGncqP4MnZ8ll8thdglRg16GDr0MHXoZOqHopaHXvE8UCAROOv7pT3+qSy65RGlpafrJT36ikpIS\nXXnllV3+fm1tc0jrcbkc+qyqUfZ4mzrbOuTxdIb0/WOJy+WQx9NodhlRgV6GDr0MHXoZOj3tZVdB\nb9iwudvtltfrDR5XVVXJ5XIFj7/zne8oMzNTdrtd06dP1+7du40qpUueumP3eJ/mjB8AgEhjWHjn\n5+erpKREklReXi632x0cMm9sbNStt96q9vZ2SdKWLVs0ZswYo0o5pZa2TjW1dKh/OsPlAABrMWzY\nfMqUKcrJyVFBQYFsNpuKioq0Zs0aORwOzZo1S9OnT9cNN9ygPn36aOLEiacdMjdC1bFheCarAQCs\nxtBr3gsXLjzpePz48cGfb775Zt18881GfvxpHa4hvAEA1hSzK6xVBcObYXMAgLXEbHhz5g0AsCrC\nm/AGAFhMzIZ3VW2zEuxx6peSaHYpAAD0SOyGd02z+qclnXZVNwAAIlFMhndLW6camzvYxxsAYEkx\nGd7BfbyZaQ4AsKAYDW+2AgUAWFdMhndKUoIS7HHKHpxmdikAAPRY2HYViyRjh6Zr5ZKrVFtzxOxS\nAADosZg885Yke3zMfnUAgMWRYAAAWAzhDQCAxRDeAABYDOENAIDFEN4AAFgM4Q0AgMUQ3gAAWAzh\nDQCAxRDeAABYDOENAIDFEN4AAFiMLRAIBMwuAgAAnDnOvAEAsBjCGwAAiyG8AQCwGMIbAACLIbwB\nALAYwhsAAIuxm12AGZYsWaKysjLZbDYVFhYqNzfX7JIsZffu3VqwYIF+8IMf6Pvf/74OHTqke++9\nVz6fTy6XS7/5zW+UmJhodpmW8Oijj2rbtm3q7OzUv/zLv+icc86hl73Q0tKi+++/X9XV1Wpra9OC\nBQs0fvx4etlLra2tuvrqq7VgwQJNmzaNPvbCpk2b9LOf/UxjxoyRJI0dO1a33XZbyHoZc2femzdv\n1v79+1VcXKzFixdr8eLFZpdkKc3NzXrooYc0bdq04GO//e1v9b3vfU8vvPCChg8frlWrVplYoXW8\n++672rNnj4qLi/X0009ryZIl9LKX1q9fr0mTJumPf/yjli1bpqVLl9LLs/D73/9eaWlpkvjv+2zk\n5eVp+fLlWr58uRYtWhTSXsZceJeWlmrmzJmSpOzsbNXX16upqcnkqqwjMTFRTz31lNxud/CxTZs2\n6YorrpAkXX755SotLTWrPEuZOnWqHn/8cUlSv3791NLSQi97ac6cObr99tslSYcOHVJWVha97KWK\nigrt3btXl112mST++w6lUPYy5sLb6/UqIyMjeOx0OuXxeEysyFrsdruSkpJOeqylpSU49JOZmUk/\nz1B8fLySk5MlSatWrdL06dPp5VkqKCjQwoULVVhYSC976ZFHHtH9998fPKaPvbd3717dcccdmjdv\nnt5+++2Q9jImr3mfiNVhQ4t+9ty6deu0atUqPfvss5o9e3bwcXrZc3/605+0a9cu3XPPPSf1j16e\nmT//+c8699xzNXTo0FM+Tx/P3IgRI3TnnXfqG9/4hiorKzV//nz5fL7g82fby5gLb7fbLa/XGzyu\nqqqSy+UysSLrS05OVmtrq5KSknT48OGThtRxehs3btT//M//6Omnn5bD4aCXvbRz505lZmZq4MCB\nmjBhgnw+n1JSUuhlD23YsEGVlZXasGGDPv/8cyUmJvI32UtZWVmaM2eOJGnYsGHq37+/duzYEbJe\nxtyweX5+vkpKSiRJ5eXlcrvdSk1NNbkqa7v44ouDPX3ttdd0ySWXmFyRNTQ2NurRRx/Vk08+qfT0\ndEn0sre2bt2qZ599VtLRS2PNzc30sheWLVum1atXa+XKlZo7d64WLFhAH3vppZde0jPPPCNJ8ng8\nqq6u1rXXXhuyXsbkrmKPPfaYtm7dKpvNpqKiIo0fP97skixj586deuSRR3Tw4EHZ7XZlZWXpscce\n0/3336+2tjYNGjRIDz/8sBISEswuNeIVFxfriSee0MiRI4OPLV26VL/61a/oZQ+1trbql7/8pQ4d\nOqTW1lbdeeedmjRpku677z562UtPPPGEBg8erK997Wv0sReampq0cOFCNTQ0qKOjQ3feeacmTJgQ\nsl7GZHgDAGBlMTdsDgCA1RHeAABYDOENAIDFEN4AAFgM4Q0AgMXE3CItQKw6cOCArrzySp133nkn\nPX7ppZfqtttuO+v337Rpk5YtW6YVK1ac9XsBOD3CG4ghTqdTy5cvN7sMAGeJ8AagiRMnasGCBdq0\naZOOHDmipUuXauzYsSorK9PSpUtlt9tls9n0b//2bxo9erT27dunRYsWye/3q0+fPnr44YclSX6/\nX0VFRdq1a5cSExP15JNPKiUlxeRvB0QfrnkDkM/n05gxY7R8+XLNmzdPv/3tbyVJ9957rx544AEt\nX75ct9xyi/793/9dklRUVKRbb71Vzz//vL773e/q1VdflXR0O8l//dd/1cqVK2W32/XWW2+Z9p2A\naMaZNxBDampqdNNNN5302D333CNJ+trXviZJmjJlip555hk1NDSourpaubm5kqS8vDz94he/kCRt\n375deXl5kqSrrrpK0tFr3qNGjVL//v0lSQMGDFBDQ4PxXwqIQYQ3EENOd837xJWSbTabbDZbl89L\nR4fIvyw+Pj4EVQLoDsPmACRJ7777riRp27ZtGjdunBwOh1wul8rKyiRJpaWlOvfccyUdPTvfuHGj\nJOmVV17Rf/3Xf5lTNBCjOPMGYsiphs2HDBkiSfrggw+0YsUK1dfX65FHHpEkPfLII1q6dKni4+MV\nFxenBx98UJK0aNEiLVq0SC+88ILsdruWLFmiTz/9NKzfBYhl7CoGQOPGjVN5ebnsdv7/PGAFDJsD\nAGAxnHkDAGAxnHkDAGAxhDcAABZDeAMAYDGENwAAFkN4AwBgMYQ3AAAW8/8BG+xjROQrK2wAAAAA\nSUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7feea84c7ba8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "jlZIDj8VqjRz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Saving summaries on disk and working with the TensorBoard"
      ]
    },
    {
      "metadata": {
        "id": "8I-1KuYKdOZQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Similarly, eager comes with its own implementation of the summaries. In order to work with them, we first define a writer on disk:"
      ]
    },
    {
      "metadata": {
        "id": "6X1lEPLoNPu2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "writer = tf.contrib.summary.create_file_writer('tmp')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "kLhIAhX9NUN1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In order to save a summary, we need to select the writer as the default one, and tell TF that we want to save summaries everytime they are computed:"
      ]
    },
    {
      "metadata": {
        "id": "ocbhuGvPNXOz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "with writer.as_default():\n",
        "  with tf.contrib.summary.always_record_summaries():\n",
        "      tf.contrib.summary.scalar('scalar_value', 0.5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9US2mkrCqwwc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Alternatively, we can instruct TF to save summaries only every given number of steps:"
      ]
    },
    {
      "metadata": {
        "id": "zDCeCkRoeCHJ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "with writer.as_default():\n",
        "  with tf.contrib.summary.record_summaries_every_n_global_steps(5):\n",
        "      tf.contrib.summary.scalar('scalar_value', 0.5) # Will only save every 5 steps"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GHB-AjS-rGVP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The 'global step' is a variable inside TF that keeps track of the iterations of our optimization algorithm:"
      ]
    },
    {
      "metadata": {
        "id": "BlhQBB4krwfa",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "28f9eb37-4582-489e-99bb-f0c9f65ed613",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043452397,
          "user_tz": -60,
          "elapsed": 767,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "tf.train.get_global_step()"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Variable 'global_step:0' shape=() dtype=int64, numpy=0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "metadata": {
        "id": "vC48OlGGsDmH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Note that the variable is currently set to 0. If we want to update it correctly, we need to provide the global step during the optimization cycle:"
      ]
    },
    {
      "metadata": {
        "id": "ym4CkTUjsLCz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "22394fab-d4f1-4fe2-bf13-f23465f560ab",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043454266,
          "user_tz": -60,
          "elapsed": 533,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "opt.apply_gradients(loss_grad(net, xb, yb), global_step=tf.train.get_or_create_global_step()) # Will apply gradients AND increase the step by one\n",
        "tf.train.get_global_step()"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Variable 'global_step:0' shape=() dtype=int64, numpy=1>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "metadata": {
        "id": "7Urz86nrsVXi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Alternatively, we can provide our own global step to the summary operation. This is particularly easy with eager execution enabled because we can work with standard int64 values:"
      ]
    },
    {
      "metadata": {
        "id": "eKet69pxsfDw",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "with writer.as_default():\n",
        "  with tf.contrib.summary.record_summaries_every_n_global_steps(5):\n",
        "      tf.contrib.summary.scalar('scalar_value', 0.5, step=4) # This will save the value on disk"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FVyM175UNRe8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In the following example, we extend again our optimization routine to save the loss value on disk at every iteration:"
      ]
    },
    {
      "metadata": {
        "id": "8DzgeRJQYc-F",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = SingleHiddenLayerNetwork()\n",
        "opt = tf.train.AdamOptimizer(learning_rate=0.01)\n",
        "\n",
        "with writer.as_default():\n",
        "    with tf.contrib.summary.always_record_summaries():\n",
        "\n",
        "        # Loop over the epochs\n",
        "        for epoch in range(50):\n",
        "\n",
        "            # For each epoch we shuffle the dataset\n",
        "            for (xb, yb) in tfe.Iterator(train_dataset.shuffle(1000).batch(32)):\n",
        "                \n",
        "                tf.contrib.summary.scalar('loss_value', loss(net, xb,yb))\n",
        "                opt.minimize(lambda: loss(net, xb,yb), global_step=tf.train.get_or_create_global_step())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9do0tqfrePJ9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now launch the tensorboard to visualize the loss:"
      ]
    },
    {
      "metadata": {
        "id": "kdgUXXPueSKT",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 3
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "outputId": "2cc49117-020f-45d8-e144-5bf46caf894c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1519213043159,
          "user_tz": -60,
          "elapsed": 17899,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "!tensorboard --logdir=tmp"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\r\n",
            "  from ._conv import register_converters as _register_converters\n",
            "TensorBoard 1.6.0 at http://eacb5bd1a82e:6006 (Press CTRL+C to quit)\n",
            "^C\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "XrrWJP-2J-JQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If you are running this notebook from a local machine, you can navigate to the address above to visualize the training details in the TensorBoard itself:"
      ]
    },
    {
      "metadata": {
        "id": "rKv8T0fTddL-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![TensorBoard (screenshot)](https://iaml.it/screenshots/tensorboard.png)"
      ]
    },
    {
      "metadata": {
        "id": "C9PxnV3x_EJo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Advanced features"
      ]
    },
    {
      "metadata": {
        "id": "rdTN7c_k_GsN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Enabling GPU execution"
      ]
    },
    {
      "metadata": {
        "id": "QtlURQ4M_vQX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "With eager execution enabled, computation on the GPU is not enabled by default (if a GPU is present). We can inspect if we have a GPU on the system easily:"
      ]
    },
    {
      "metadata": {
        "id": "3K3Qf5nn_FTX",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2310e3c6-b512-4902-e9c7-79ee24fcc1a7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043461605,
          "user_tz": -60,
          "elapsed": 509,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "tfe.num_gpus()"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "metadata": {
        "id": "VGGNOsfTAgky",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In order to enable the GPU, we can specify a device explicitly for each operation that we want to run on that device:"
      ]
    },
    {
      "metadata": {
        "id": "7OOl75ww_F6h",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.device(\"/gpu:0\"):\n",
        "  net(X_train[0:1, :])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Az0i2ydGBBcS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Alternatively, we can move the data to the GPU before running the computation:"
      ]
    },
    {
      "metadata": {
        "id": "e3Ce7zhOA8DG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "724596d6-bb45-4b1c-e5d2-a25e1369d720",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1519222839412,
          "user_tz": -60,
          "elapsed": 594,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = net.gpu()\n",
        "net(tf.constant(X_train[0:1, :]).gpu())"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: id=78996, shape=(1, 3), dtype=float64, numpy=array([[-1.14483888,  0.18914156,  0.74425629]])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "metadata": {
        "id": "kcLGEpoXUEWk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Given a variable `x` on the GPU, we can perform the inverse operation similarly:\n",
        "\n",
        "\n",
        "```python\n",
        "x = x.cpu()\n",
        "```\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "b_I-_yQnB_Cw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Saving and reloading"
      ]
    },
    {
      "metadata": {
        "id": "eOUXvR-ZCDUQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Eager provides a simple interface for saving variables on disk:"
      ]
    },
    {
      "metadata": {
        "id": "xet5_u5hCAoD",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5e336b0c-cd2e-4902-c0f3-1b9762e59159",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043481286,
          "user_tz": -60,
          "elapsed": 527,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "checkpointer = tfe.Checkpoint(W=W)\n",
        "checkpointer.save('tmp/')\n",
        "checkpointer.restore('tmp/')"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.contrib.eager.python.checkpointable_utils.NameBasedSaverStatus at 0x7feea612deb8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "metadata": {
        "id": "EOkoabdJCmNo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In order to save a comprehensive snapshot, we can save all variables of the model together with the optimizer's state, e.g.:"
      ]
    },
    {
      "metadata": {
        "id": "Xfe2oTN7CUdM",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d3086f64-49e1-4595-d82d-b3311b1a8ae4",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043574317,
          "user_tz": -60,
          "elapsed": 509,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "checkpointer = tfe.Checkpoint(net=net,opt=opt, global_step=tf.train.get_or_create_global_step())\n",
        "checkpointer.save('tmp/')"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'tmp/-1'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "metadata": {
        "id": "wNpMQI2DvTiG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We modify one again our training routine, this time by saving all variables at the end of every epoch:"
      ]
    },
    {
      "metadata": {
        "id": "qtTsuSvveI0X",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "net = SingleHiddenLayerNetwork()\n",
        "opt = tf.train.AdamOptimizer(learning_rate=0.01)\n",
        "\n",
        "checkpointer = tfe.Checkpoint(net=net,opt=opt, global_step=tf.train.get_or_create_global_step())\n",
        "\n",
        "for epoch in range(50):\n",
        "  for (xb, yb) in tfe.Iterator(train_dataset.shuffle(1000).batch(32)):\n",
        "    opt.minimize(lambda: loss(net, xb,yb))\n",
        "   \n",
        "  checkpointer.save('tmp/')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Rw9R1zHYvkM0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Eager provides a method to restore the latest checkpoint from the disk:"
      ]
    },
    {
      "metadata": {
        "id": "UJ3VAquGedDa",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "output_extras": [
            {
              "item_id": 1
            }
          ],
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4edd8824-4019-4c84-f22f-ee6fe0287d5c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1521043657225,
          "user_tz": -60,
          "elapsed": 483,
          "user": {
            "displayName": "Simone Scardapane",
            "photoUrl": "//lh6.googleusercontent.com/-ZK0nd0iCe7w/AAAAAAAAAAI/AAAAAAAAA_o/pNhWxdOsGTI/s50-c-k-no/photo.jpg",
            "userId": "115632300554726808471"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "checkpointer.restore(tf.train.latest_checkpoint('tmp/'))"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.contrib.eager.python.checkpointable_utils.CheckpointLoadStatus at 0x7feea62c4748>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "metadata": {
        "id": "fVtZn99KDDSG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Final remarks"
      ]
    },
    {
      "metadata": {
        "id": "0jxXEvJEDGhN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In this tutorial, we have seen that developing at a low-level with the new eager execution can be easier (and more simple to debug) that with the standard low-level interface. Since the module is still experimental, some advanced features (such as distributed training of graphs) are not yet available when running with the eager execution enabled, and the interface is still prone to minor changes. We refer to the [official guide](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/g3doc/guide.md) for more information on all these aspects."
      ]
    }
  ]
}
